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Program description . 


Fluency Formula™ 

Fluency Formula™ is a supplemental curriculum designed to 
promote reading fluency for first- through sixth-grade students. 
The program emphasizes automatic recognition of words, 
decoding accuracy, and oral expressiveness as the foundation 
for building reading fluency. A daily 10- to 15-minute lesson is 
delivered in the classroom. Students participate in whole-class. 


small-group, and individual practice activities using workbooks, 
read-aloud anthologies, library books, fluency activity cards, and 
audio CDs. The curriculum encourages at-home practice and 
includes a Fluency Formula™ Assessment System, which allows 
teachers to assess student fluency using one-minute grade-level 
passages and a timer. 



RGSGarch One study of Fluency Fornnula™ met the What Works Clear- of evidence for Fluency Formula™ to be small for fluency and 

inghouse (WWC) evidence standards. The study findings are comprehension. No studies that met WWC evidence standards 

based on data from 128 second-grade students in two suburban with or without reservations addressed aiphabetics or generai 

schooi districts in the Northeast.'' The WWC considers the extent reading achievement. 



EffGCtiVGnGSS 



Fluency Formula™ was found to have potentially positive effects on fluency and potentially negative effects on comprehension. 



Rating of effectiveness 
Improvement index^ 



Aiphabetics 


Fluency 


Comprehension 


General reading 
achievement 


na 


Potentially positive 


Potentially negative 


na 




effects 


effects 




na 


Average: 


Average: 


na 




+10 percentile points 


-11 percentile points 





na = not applicable 



1. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

2. These numbers show the improvement indices based on the single finding in each domain in the study. 
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Additional program Developer and contact 

information Fluency Formula™ is distributed by Scholastic. Web: www. 

scholastic.com/fluencvformula . Telephone: (800) 724-6527 
(customer service). 

Scope of use 

Fluency Formula™ has been implemented in eight states in the 
United States. 

Teaching 

Teachers use a variety of strategies during 15-minute daily 
lessons to promote students’ knowledge and skills in phonics, 
vocabulary, fluency, and comprehension. The first two days of 
each instructional week are devoted to direct instruction and 
guided practice activities for the whole class with a focus on 
fluent reading, prosody (intonation and/or vocal stress), and oral 
comprehension. During the next two days of the week, students 
reading at and above grade level practice their reading fluency 
independently, while students reading below grade level receive 
supplemental instruction in small groups with an emphasis on 
phonics and word attack skills. On the fifth day, teachers use 
the Fluency Formula™ Assessment System to identify struggling 
students and then use the assessment handbook to set instruc- 
tional goals based on student assessment scores. In addition, 
teachers monitor student fluency growth at the beginning, 
middle, and end of the school year. 

During class lessons, teachers model fluent reading and ask 
students to practice through repeated readings, reading with 



partners, and together in whole class (choral) reading. Students 
practice fluency, vocabulary, and comprehension individually 
using a program workbook and read library books while listening 
to an audio CD version of the book. Instructional practices also 
include review of sight word fluency and drills for reading speed 
and accuracy. Students can track their fluency progress on a 
class chart and receive bookmarks and certificates as rewards 
for progress. Students are encouraged to take the library books 
and workbooks home and read for at least 20-30 minutes daily 
at home or at school. 

Professional development for teachers includes the Fluency 
Formula™ Professional Guide as well as online courses through 
Scholastic’s “Scholastic Red” division, which include teaching 
simulations, structured feedback, information on research and 
theory, videos modeling instruction by expert practitioners, and 
materials that can be used in the classroom. 

Cost 

The Fluency Formula™ kit for one grade level costs $625 and 
includes introductory materials, four student workbooks, four 
copies each of 24 library books and six audio CDs, fluency activ- 
ity cards, original leveled reading passages, the Fluency For- 
mula™ Assessment System with a student timer, and a fluency 
flip chart. The cost for six Fluency Formula™ Kits (one per grade 
for grades 1-6), a principal’s guide, and a facilitator handbook is 
$5,775. The cost of individual materials and courses is available 
on the publisher’s website. 



Research Met evidence standards 

One study examined the effects of Fluency Formula™. The study 
(Sivin-Kachala & Bialo, 2005) was a randomized controlled trial 
that met WWC evidence standards. The study began with 252 
second-grade students from 12 classrooms in two suburban 
school districts in the Northeast. However, because the 
intervention targets lower-achieving students, the study authors 



focused only on students scoring below the 75th percentile 
on the Oral Fluency Assessment at pretest. The final analysis 
sample involved 128 students who scored below this level. 

The intervention group received the school’s standard read- 
ing/English language arts materials and the Fluency Formula™ 
supplemental curriculum. The comparison group received just 
the school’s standard reading/English language arts materials. 
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Research (continued) 



Effectiveness 



WWC Intervention Report 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain as 
small or moderate to large (see the What Works Clearinghouse 
Extent of Evidence Categorization Scheme) . The extent of 
evidence takes into account the number of studies and the 



total sample size across the studies that met WWC evidence 
standards with or without reservations.® 

The WWC considers the extent of evidence for Fluency For- 
mula™ to be small for fluency and comprehension. No studies 
that met WWC evidence standards with or without reservations 
addressed alphabetics or general reading achievement. 



Findings 

The WWC review of interventions for beginning reading 
addresses student outcomes in four domains: alphabetics, 
fluency, comprehension, and general reading achievement."* The 
Sivin-Kachala & Bialo (2005) study addressed outcomes in the 
fluency and comprehension domains.® The findings below pres- 
ent the authors’ and the WWC-calculated estimates of the size 
and statistical significance of the effects of Fluency Formula™ 
on students’ performance. 

Fluency. The Sivin-Kachala & Bialo (2005) study findings for 
fluency are based on the performance of Fluency Formula™ stu- 
dents and comparison students on the Edformation Oral Fluency 
Assessment (OFA). The WWC found no statistically significant 
difference between the groups, however, the effect was positive 
and substantively important according to WWC criteria (that is, 
an effect size of at least .25).® 



Comprehension. The Sivin-Kachala & Bialo (2005) study find- 
ings for comprehension are based on the performance of Fluency 
Formula™ students and comparison students on the Woodcock- 
Johnson III Tests of Achievement: Passage Comprehension 
subtest. The WWC did not find a statistically significant difference 
between the groups. However, the effect was negative and sub- 
stantively important (that is, an effect size of at least -.25).^ 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings,® the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Intervention Rating Scheme) . 



3. The Extent of Evidence categorization was developed to tell readers how much evidence was used to determine the intervention rating, focusing on the 
number and size of studies. Additionai factors associated with a related concept, external validity, such as the students’ demographics and the types of 
settings in which studies took place, are not taken into account for the categorization. 

4. For definitions of the domains, seethe i III; i ; : : . 

5. The study also examined data in the alphabetics domain, but complete data were not available in the study report or from the study authors. 

6. The study only reported intervention effects by high- and low-ability student groups, based on the OFA pretest score. These results are shown in Appen- 
dix A4. The overall effects presented here were computed by the WWC. The study authors reported both raw and normal curve equivalent (NOE) scores. 
The WWC used the raw score results. 

7. The study reported intervention effects on comprehension by each district in the study and for low-ability students. The overall effects presented here 
were computed by the WWC. The results for low-ability students are shown in Appendix A4. 

8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within class- 
rooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See the Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of Fluency Formula™, corrections for clustering were 
needed. 
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The WWC found Fluency 
Formula™ to have 
potentially positive 
effects for fluency and 
potentially negative effects 
for comprehension 



References 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and an 
average improvement index across studies (see Technical Details 
of WWC-Conducted Computations^ . The improvement index rep- 
resents the difference between the percentile rank of the average 
student in the intervention condition versus the percentile rank 
of the average student in the comparison condition. Unlike the 
rating of effectiveness, the improvement index is based entirely 
on the size of the effect, regardless of the statistical significance 
of the effect, the study design, or the analyses. The improvement 
index can take on values between -50 and +50, with positive 
numbers denoting results favorable to the intervention group. 



The improvement index for fluency is +10 percentile points for 
the fluency outcome in the single study reviewed. The improve- 
ment index for comprehension is -11 percentile points for the 
comprehension outcome in the single study. 

Summary 

The WWC reviewed one study on Fluency Formula™. This study 
met WWC evidence standards. Based on this single study, the 
WWC found potentially positive effects in the fluency domain 
and potentially negative effects in the comprehension domain. 
The evidence presented in this report may change as new 
research emerges. 



Met WWC evidence standards 

Sivin-Kachala, J., & Bialo, E. (2005). Fluency Formula second 
grade study. Long Island, New York 2003-2004: Evaluation 
research on the effectiveness of Fluency Formula. Retrieved 
from Scholastic Education Web site: http://teacher.scholastic. 
com/products/fluencvformula/pdfs/FF EffectivessReport.pdf 



Additional source: 

Sivin-Kachala, J., & Bialo, E. (2005). Fluency Formula second 
grade study. Long Island, New York 2003-2004: A sum- 
mary of the effectiveness research for Fluency Formula. 
Retrieved from Scholastic Education Web site: http:// 
teacher.scholastic.com/products/fluencvformula/pdfs/ 

FF EffectivenessSummarv.pdf 



For more information about specific studies and WWC caicuiations, piease see the WWC Fluency Formula™ 
Technicai Appendices . 
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Appendix A1 Study characteristics: Sivin-Kachaia & Biaio, 2005 (randomized controiied triai) 



Characteristic 


Description 


Study citation 


Sivin-Kachaia, J., & Biaio, E. (2005). Fluency Formula second grade study, Long Island, New York 2003-2004: Evaluation research on the effectiveness of Fluency Formula. 
Retrieved from Scholastic Education Web site: httD://teacher.scholastic.com/oroducts/fluencvformula/Ddfs/FF EffectivessReoort.odf 


Participants 


The study began with 252 second-grade students (127 in the intervention condition) from 12 classrooms (six classrooms in each condition). Participants came from three 
schools in one district and two schools in a second district (personal communication). Classrooms were matched on students’ reading ability, demographic variables, and 
teacher characteristics and then randomly assigned to study conditions. Because Fluency Formula™ does not target high-performing students, analyses were limited to the 
143 students scoring at or below the 75th percentile on the Edformation Oral Fluency Assessment (OFA) pretest. Fifteen additional students were removed from analyses: one 
student who moved from the comparison to intervention group, eight students who exited the program, four students who missed at least half the intervention lessons, and 
two students with a large number of absences. The final analyses were based on data from 66 intervention students and 62 comparison students. According to data provided 
in the study, the remaining students (analysis sample) from the intervention and comparison groups were comparable on pretest achievement measures (OFA).^ 


Setting 


The study was conducted in five schools^ across two suburban school districts in the northeastern United States. 


Intervention 


Fluency Formula™ was delivered as a supplement to the participating schools' standard reading/English language arts curriculum. Intervention group teachers followed the 
six-unit curriculum sequence in the Fluency Formula™ Professional Guide. Students classified as “low initial ability” (based on the pretest of oral fluency) received four days a 
week of Fluency Formula™ instruction (two days of whole-class instruction plus two days of small-group instruction). Students classified as “high initial ability” received two 
days of whole-class Fluency Formula™ instruction. Once a week, students received a 15-minute take-home assignment. 


Comparison 


The comparison group students received the participating schools' standard reading/English language arts curriculum with no supplemental materials or instruction. The 
comparison group teachers were not exposed to the Fluency Formula™ materials or professional development. 


Primary outcomes 
and measurement 


The primary outcome measure in the fluency domain is the Edformation Oral Fluency Assessment (OFA) and the primary outcome in the comprehension domain is the 
Woodcock-Johnson III Tests of Achievement: Passage Comprehension subtest (see Appendixes A2.1 and A2.2 for more detailed descriptions of outcome measures).^ 


Teacher training 


Intervention group teachers attended one after-school professional development session (about two and a half hours) that presented the theoretical basis, components, and 
implementation of the program. Teachers followed the instructional sequence detailed in the Fluency Formula™ Professional Guide. 



1 . The authors also used the Woodcock-Johnson III Basic Reading cluster score as a pretest and indicated that there were no statistically significant differences between the treatment and com- 
parison students. These data were provided by the study authors in personal communication. For one outcome (comprehension), the authors analyzed the two (unidentified) districts separately. 
In that analysis, the comparison group in one of the districts scored significantly lower than the intervention group on pretest. The WWC, however, considered the pooled data, rather than data 
for the separate districts. 

2. This information was provided by the study authors in personal communication. 

3. The study also included student outcomes in the alphabetics domain, but those outcomes were not reviewed for rating purposes because the data were not reported in the study. 
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Appendix A2.1 Outcome measures in the fluency domain 



Characteristic 


Description 


Edformation Oral Fluency 
Assessment (OFA) 


This test measures the number of words correct per minute (WCPM) that students read using three brief grade-ievel passages (200-word minimum). These passages inciude 
both fiction and nonfiction text. The norms for this test are updated by Edformation each schooi year^ (as cited in Sivin-Kachaia & Biaio, 2005). 



1. The study authors reported both raw and normal curve equivalent (NCE) scores. The WWC used the raw score results. 



Appendix A2.2 Outcome measures in the comprehension domain 



Characteristic 


Description 


Woodcock-Johnson III Tests 
of Achievement: Passage 
Comprehension subtest 


This standardized test measures chiidren’s abiiity to match words and pictures of objects the words stand for and to identify missing key words in the reading passage (as 
cited in Sivin-Kachaia & Biaio, 2005). 
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Appendix A3.1 



Summary of study findings inciuded in the rating for the fluency domain^ 









Authors’ findings from the study 














Mean outcome^ 
(standard deviation^) 




WWC calculations 








Sample size 


Fluency 


Mean difference'* 


Statistical 






Study 


(classrooms/ 


Formula™ (FF) Comparison 


(FF- 


significance^ 


Improvement 


Outcome measure 


sample 


students) 


group group 


comparison) 


Effect size® (at a = 0.05) 


index^ 



Sivin-Kachala & Bialo, 2005 (randomized controlled trial)® 








Oral Fluency Assessment (OFA) Grade 2 12/128 84.85 78.62 6.24 

(19.65) (27.88) 


0.26 


ns 


-FlO 


Domain average® for fluency (Sivin-Kachala & Bialo, 2005) 


0.26 


ns 


-FlO 



ns = not statistically significant 

1 . This appendix reports the findings considered for the effectiveness rating and the average improvement indices. Subgroup findings from the same study are not included in these ratings, but are reported in Appendix A4. 

2. The means for the intervention and comparison groups were calculated by the WWC by combining initial fluency ability subgroup means (see Technical Details of WWC-Conducted Computationsf . The reported intervention group mean 
equals the comparison group mean plus the mean difference. 

3. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are; a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard 
deviations for the combined intervention and comparison groups were calculated by the WWC based on subgroup standard deviations. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The mean difference was calculated using the difference-in-difference (gain scores) approach; see 
Technical Details of WWC-Conducted Computations . 

5. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and -i-50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clus- 
tering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Sivin-Kachala & Bialo (2005), a correction 
for clustering was needed, so the significance levels may differ from those reported in the original study. 

9. This row provides the study average, which in this instance, is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated 
from the average effect size. 
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Appendix A3.2 Summary of study findings inciuded in the rating for the comprehension domain^ 









Authors’ findings from the study 














Mean outcome^ 
(standard deviation^) 




WWC calculations 








Sample size 


Fluency 


Mean difference'* 


Statistical 






Study 


(classrooms/ 


Formula™ (FF) Comparison 


(FF- 


significance^ 


Improvement 


Outcome measure 


sample 


students) 


group group 


comparison) 


Effect size® (at a = 0.05) 


index^ 



Sivin-Kachala & Bialo, 2005 (randomized controlled trial)® 








Woodcock-Johnson III: Passage Grade 2 12/128 477.46 480.82 -3.36 

Comprehension subtest (10.88) (13.05) 


-0.28 


ns 


-11 


Domain average® for comprehension (Sivin-Kachala & Bialo, 2005) 


-0.28 


ns 


-11 



ns = not statistically significant 

1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices. 

2. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The reported intervention group mean equals the comparison group mean plus the mean difference. 

3. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are; a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. Standard 
deviations for the combined intervention and comparison groups were calculated by the WWC based on subgroup standard deviations from two districts. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The mean difference was calculated using the difference-in-difference (gain scores) approach; see 
Technical Details of WWC-Conducfed Computations . 

5. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and -t-50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clus- 
tering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Sivin-Kachala & Bialo (2005), a correction 
for clustering was needed, so the significance levels may differ from those reported in the original study. 

9. This row provides the study average, which, in this instance, is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated 
from the average effect size. 
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Appendix A4 Summary of subgroup findings for the fluency domain^ 









Authors’ findings from the study 














Mean outcome^ 
(standard deviation^) 




WWC calculations 








Sample size 


Fluency 


Mean difference'* 


Statistical 






Study 


(classrooms/ 


Formula™ (FF) Comparison 


(FF- 


significance^ 


Improvement 


Outcome measure 


sample 


students) 


group group 


comparison) 


Effect size® (at a = 0.05) 


index^ 



Oral Fluency Assessment (OFA) 


Grade 2 


Sivin-Kachala & Bialo, 2005 (randomized controlled trial) — Low-ability subgroup® 

12/72 74.28 60.83 13.45 0.66 

(17.53) (22.40) 


ns 


+25 






Sivin-Kachala & Bialo, 2005 (randomized controlled trial) — High-ability subgroup® 






Oral Fluency Assessment (OFA) 


Grade 2 


12/56 98.63 101.67 -3.04 -0,21 

(14.43) (13.79) 


ns 


-11 



ns = not statistically significant 

1 . This appendix presents ability subgroup findings for measures that fall in the fluency domain. Total group scores were used for rating purposes and are presented in Appendix A3.1 . 

2. The intervention group mean equals the comparison group mean plus the mean difference. 

3. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are; a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The mean difference was calculated using the difference in difference (gain scores) approach; see 
Technical Details of WWC-Conducted Computations . 

5. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition versus the percentile rank of the average student in the comparison condition. The improvement index 
can take on values between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the clus- 
tering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Sivin-Kachala & Bialo (2005), a correction 
for clustering was needed, so the significance levels may differ from those reported in the original study. 
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Appendix A5.1 Fluency Formula™ rating for the fluency domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaily positive, mixed, no discernibie effects, potentialiy negative, or negative.'' 

For the outcome domain of fluency, the WWC rated Fluency Formula™ as having potentiaily positive effects. It did not meet the criteria for the positive effects 
because there was oniy one study. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, negative effects) were not considered 
because Fluency Formula™ was assigned the highest applicable rating. 

Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. The one study reviewed showed substantively important positive effects. 

and 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. No study showed statistically significant or substantively important negative effects or indeterminate effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Only one study met WWC evidence standards. 

and 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies showed statistically significant or substantively important negative effects. 



1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A5.2 Fluency Formula™ rating for the comprehension domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaily positive, mixed, no discernibie effects, potentialiy negative, or negative.'' 

For the outcome domain of comprehension, the WWC rated Fluency Formula™ as having potentiaiiy negative effects. It did not meet the criteria for other ratings 
(positive effects, potentiaily positive effects, mixed effects, no discernibie effects, and negative effects) because the single study that met WWC standards showed 
substantively important negative effects. 

Rating received 

Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence 

• Criterion 1; At least one study showing a statistically significant or substantively important negative effect. 

Met. The one study reviewed showed substantively important negative effects. 

and 

• Criterion 2: No studies showing a statistically significant or substantively important positive effect, or more studies showing statistically significant or substantively 
important negative effects than showing statistically significant or substantively important pos/f/ve effects. 

Met. No studies showed statistically significant or substantively important positive effects. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Only one study met WWC evidence standards. 

and 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Not met. One study showed substantively important negative effects. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. No studies showed statistically significant or substantively important positive effects. 

and 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. One study showed substantively important negative effects. 



(continued) 
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Appendix A5.2 Fluency Formula™ rating for the comprehension domain (continued) 



Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. No studies showed statistically significant or substantively important positive effects. 

or 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Not met. One study showed substantively important negative effects and no study showed indeterminate effects. 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: None of the studies shows a statistically significant or substantively important effect, either pos/f/Ve or negative. 

Not met. One study showed substantively important negative effects. 

1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A6 Extent of evidence rating by domain 



Outcome domain 


Number of studies 


Schoois 


Sampie size 


Students 


Extent of evidence^ 


Alphabetics 


0 


0 




0 


na 


Comprehension 


1 


5 




128 


Small 


Fluency 


1 


5 




128 


Small 


General reading achievement 


0 


0 




0 


na 



na = not applicable/not studied 

1 . A rating of “moderate to large” requires at least two studies and two schools across studies in one domain and a total sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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